Skip to content

fix(chunker): respect contextualized table token limits - #690

Draft
pablopupo wants to merge 2 commits into
docling-project:mainfrom
pablopupo:fix/3428-contextualized-chunk-limit
Draft

fix(chunker): respect contextualized table token limits#690
pablopupo wants to merge 2 commits into
docling-project:mainfrom
pablopupo:fix/3428-contextualized-chunk-limit

Conversation

@pablopupo

@pablopupo pablopupo commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
  • Apply the HybridChunker metadata-adjusted budget when splitting Markdown table rows.
  • Count exact combined fragments at token boundaries and preserve table content when an oversized prefix is split.
  • Add regressions for contextualized overflow, boundary tokenization, and prefix-to-body reconstruction.

Resolves docling-project/docling#3428

Verification

  • pytest --cov=docling_core --cov-report=xml test: 566 passed, 6 skipped
  • ruff format --check docling_core test, ruff check docling_core test, and mypy docling_core test
  • Supplied PDF: maximum contextualized chunk 8,189 / 8,192; both tables reconstructed byte-for-byte

@pablopupo
fix(chunker): respect contextualized table token limits
afb092a 
Signed-off-by: Pablo Pupo <pablofpupo23@gmail.com>
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @pablopupo, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

Protection Waiting on
🔴 Require two reviewer for test updates 👀 reviews
🟢 Enforce conventional commit —

🔴 Require two reviewer for test updates

Waiting for

  • #approved-reviews-by >= 2
This rule is failing.

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2
Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.35294% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
docling_core/transforms/chunker/line_chunker.py 81.25% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@pablopupo
test(chunker): cover concatenated token boundary
c95ce2f 
Signed-off-by: Pablo Pupo <pablofpupo23@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[MarkdownTableSerializer] Chunk overflow with landscape tables in non-landscape mode

1 participant

@pablopupo